Speech Recognition on English-Mandarin Code-Switching Data using Factored Language Models - with Part-of-Speech Tags, Language ID and Code-Switch Point Probability as Factors pdfsubject=Multilingual Speech Recognition

Code-switching is defined as ”the alternate use of two or more languages in the same utterance or conversation” [1]. CS is a wide-spread phenomenon in multilingual communities, where multiple languages are concurrently used in a conversation. For automatic speech recognition (ASR), particularly intra-sentential code-switching poses an interesting challenge due to the multilingual context for language modeling and the acoustic model. The statistical estimation of n-grams at code-switch points (CSP) is poor for common, word-based n-gram language models. Our work investigates the application of additional features for ASR on MandarinEnglish speech data containing intra-sentential CS. First we explore the use of various features for multilingual language modeling by predicting language and CSPs. We investigate the text based prediction of the language of a word, utilizing language, part-of-speech (POS) tags and occurrence features of previous words in the current utterance. The results of our language prediction experiments are used to predict CSPs. We can reach an improvement compared to random predictions of CSPs. The results of our experiments to predict language and CSPs indicate, that it is useful to add language and POS tag features in the multilingual language model applied on CS data. Consequently, we include language identification (LID), POS tags, and a CSP probability class into the language model. We utilize factored language models to incorporate these features. N-best rescoring is applied to use factored language models for speech recognition. We obtain the highest performance utilizing factored language models with LID, POS tags and words as features. We evaluate our results utilizing a mixed error rate (MER). The MER is defined as the character error rate (CER) for Mandarin and the word error rate (WER) for English. Using our best performing factored language model, we improve the MER performance from 59.1% to 58.4% on the development set and from 60.8% to 59.8% on the evaluation set.